{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "modeling_tutorial.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lvapeab/nmt-keras/blob/master/examples/modeling_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "62qtTHgCJcb9"
      },
      "source": [
        " # Model definition in NMT-Keras\n",
        "\n",
        "\n",
        "In this module, we are going to create an encoder-decoder model with:\n",
        "* A bidirectional GRU encoder and a GRU decoder\n",
        "* An attention model \n",
        "* The previously generated word feeds back de decoder\n",
        "* MLPs for initializing the initial RNN state\n",
        "* Skip connections from inputs to outputs\n",
        "* Beam search.  \n",
        "\n",
        "We first setup the machine."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AzPGNL96JcvI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "ffe33a0c-8510-41a2-c9a2-6dfc793d8b69"
      },
      "source": [
        "!pip install update pip\n",
        "!pip uninstall -y kapre keras tensorflow-probability albumentations datascience h5py keras-nightly # Avoid crashes with pre-installed packages \n",
        "!git clone https://github.com/lvapeab/nmt-keras\n",
        "import os\n",
        "os.chdir('nmt-keras')\n",
        "!pip install -e .\n"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting update\n",
            "  Downloading update-0.0.1-py2.py3-none-any.whl (2.9 kB)\n",
            "Requirement already satisfied: pip in /usr/local/lib/python3.7/dist-packages (21.1.3)\n",
            "Collecting style==1.1.0\n",
            "  Downloading style-1.1.0-py2.py3-none-any.whl (6.4 kB)\n",
            "Installing collected packages: style, update\n",
            "Successfully installed style-1.1.0 update-0.0.1\n",
            "Found existing installation: kapre 0.3.5\n",
            "Uninstalling kapre-0.3.5:\n",
            "  Successfully uninstalled kapre-0.3.5\n",
            "Found existing installation: Keras 2.4.3\n",
            "Uninstalling Keras-2.4.3:\n",
            "  Successfully uninstalled Keras-2.4.3\n",
            "Found existing installation: tensorflow-probability 0.13.0\n",
            "Uninstalling tensorflow-probability-0.13.0:\n",
            "  Successfully uninstalled tensorflow-probability-0.13.0\n",
            "Found existing installation: albumentations 0.1.12\n",
            "Uninstalling albumentations-0.1.12:\n",
            "  Successfully uninstalled albumentations-0.1.12\n",
            "Found existing installation: datascience 0.10.6\n",
            "Uninstalling datascience-0.10.6:\n",
            "  Successfully uninstalled datascience-0.10.6\n",
            "Found existing installation: h5py 3.1.0\n",
            "Uninstalling h5py-3.1.0:\n",
            "  Successfully uninstalled h5py-3.1.0\n",
            "Found existing installation: keras-nightly 2.5.0.dev2021032900\n",
            "Uninstalling keras-nightly-2.5.0.dev2021032900:\n",
            "  Successfully uninstalled keras-nightly-2.5.0.dev2021032900\n",
            "Cloning into 'nmt-keras'...\n",
            "remote: Enumerating objects: 4799, done.\u001b[K\n",
            "remote: Counting objects: 100% (69/69), done.\u001b[K\n",
            "remote: Compressing objects: 100% (55/55), done.\u001b[K\n",
            "remote: Total 4799 (delta 28), reused 30 (delta 14), pack-reused 4730\u001b[K\n",
            "Receiving objects: 100% (4799/4799), 5.75 MiB | 21.82 MiB/s, done.\n",
            "Resolving deltas: 100% (3243/3243), done.\n",
            "Obtaining file:///content/nmt-keras\n",
            "Collecting keras@ https://github.com/MarcBS/keras/archive/master.zip\n",
            "  Downloading https://github.com/MarcBS/keras/archive/master.zip\n",
            "\u001b[K     - 101.2 MB 40 kB/s\n",
            "\u001b[?25hRequirement already satisfied: cloudpickle in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.3.0)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (0.16.0)\n",
            "Collecting keras_applications\n",
            "  Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)\n",
            "\u001b[K     |████████████████████████████████| 50 kB 2.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: keras_preprocessing in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.1.2)\n",
            "Collecting h5py\n",
            "  Downloading h5py-3.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 4.1 MB 9.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (3.2.2)\n",
            "Collecting multimodal-keras-wrapper\n",
            "  Downloading multimodal_keras_wrapper-3.1.6-py3-none-any.whl (124 kB)\n",
            "\u001b[K     |████████████████████████████████| 124 kB 44.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.19.5)\n",
            "Requirement already satisfied: scikit-image in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (0.16.2)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (0.22.2.post1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.15.0)\n",
            "Requirement already satisfied: tables in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (3.4.4)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.1.5)\n",
            "Collecting sacrebleu\n",
            "  Downloading sacrebleu-1.5.1-py3-none-any.whl (54 kB)\n",
            "\u001b[K     |████████████████████████████████| 54 kB 2.5 MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "  Downloading sacremoses-0.0.45-py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 42.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.4.1)\n",
            "Collecting tensorflow<2\n",
            "  Downloading tensorflow-1.15.5-cp37-cp37m-manylinux2010_x86_64.whl (110.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 110.5 MB 1.3 kB/s \n",
            "\u001b[?25hRequirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from keras@ https://github.com/MarcBS/keras/archive/master.zip->nmt-keras==0.6) (3.13)\n",
            "Collecting gast==0.2.2\n",
            "  Downloading gast-0.2.2.tar.gz (10 kB)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (0.8.1)\n",
            "Collecting tensorflow-estimator==1.15.1\n",
            "  Downloading tensorflow_estimator-1.15.1-py2.py3-none-any.whl (503 kB)\n",
            "\u001b[K     |████████████████████████████████| 503 kB 49.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (3.17.3)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (1.12.1)\n",
            "Collecting h5py\n",
            "  Downloading h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.9 MB 44.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (0.2.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (0.36.2)\n",
            "Collecting tensorboard<1.16.0,>=1.15.0\n",
            "  Downloading tensorboard-1.15.0-py3-none-any.whl (3.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.8 MB 33.6 MB/s \n",
            "\u001b[?25hCollecting numpy\n",
            "  Downloading numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 20.1 MB 1.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (3.3.0)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (0.12.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (1.1.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (1.34.1)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (57.2.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (3.3.4)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (1.0.1)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (4.6.1)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (3.7.4.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (3.5.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->nmt-keras==0.6) (1.3.1)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->nmt-keras==0.6) (2.4.7)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->nmt-keras==0.6) (2.8.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->nmt-keras==0.6) (0.10.0)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.7/dist-packages (from multimodal-keras-wrapper->nmt-keras==0.6) (0.11.1)\n",
            "Requirement already satisfied: cython in /usr/local/lib/python3.7/dist-packages (from multimodal-keras-wrapper->nmt-keras==0.6) (0.29.23)\n",
            "Collecting subword-nmt\n",
            "  Downloading subword_nmt-0.3.7-py2.py3-none-any.whl (26 kB)\n",
            "Requirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from multimodal-keras-wrapper->nmt-keras==0.6) (0.0)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->nmt-keras==0.6) (2018.9)\n",
            "Collecting portalocker==2.0.0\n",
            "  Downloading portalocker-2.0.0-py2.py3-none-any.whl (11 kB)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->nmt-keras==0.6) (1.0.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from sacremoses->nmt-keras==0.6) (4.41.1)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from sacremoses->nmt-keras==0.6) (2019.12.20)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->nmt-keras==0.6) (7.1.2)\n",
            "Requirement already satisfied: PyWavelets>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->nmt-keras==0.6) (1.1.1)\n",
            "Requirement already satisfied: pillow>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->nmt-keras==0.6) (7.1.2)\n",
            "Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->nmt-keras==0.6) (2.4.1)\n",
            "Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->nmt-keras==0.6) (2.5.1)\n",
            "Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.0->scikit-image->nmt-keras==0.6) (4.4.2)\n",
            "Requirement already satisfied: numexpr>=2.5.2 in /usr/local/lib/python3.7/dist-packages (from tables->nmt-keras==0.6) (2.7.3)\n",
            "Building wheels for collected packages: keras, gast\n",
            "  Building wheel for keras (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras: filename=Keras-2.3.1.1-py3-none-any.whl size=487514 sha256=8e520e79b6f01a770035da390bf4c47703b880362afdb4d1c617f10d5d659fe5\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-u0i33hh5/wheels/68/86/9b/290dd8e0919a4070424e29c34886fbcf85d437c53506723c08\n",
            "  Building wheel for gast (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gast: filename=gast-0.2.2-py3-none-any.whl size=7553 sha256=99239e66fe3b012329c6995662d8170122422c4500709ba63bcf90cb4e53652e\n",
            "  Stored in directory: /root/.cache/pip/wheels/21/7f/02/420f32a803f7d0967b48dd823da3f558c5166991bfd204eef3\n",
            "Successfully built keras gast\n",
            "Installing collected packages: numpy, h5py, portalocker, keras-applications, tensorflow-estimator, tensorboard, subword-nmt, sacremoses, sacrebleu, keras, gast, tensorflow, multimodal-keras-wrapper, nmt-keras\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.19.5\n",
            "    Uninstalling numpy-1.19.5:\n",
            "      Successfully uninstalled numpy-1.19.5\n",
            "  Attempting uninstall: tensorflow-estimator\n",
            "    Found existing installation: tensorflow-estimator 2.5.0\n",
            "    Uninstalling tensorflow-estimator-2.5.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.5.0\n",
            "  Attempting uninstall: tensorboard\n",
            "    Found existing installation: tensorboard 2.5.0\n",
            "    Uninstalling tensorboard-2.5.0:\n",
            "      Successfully uninstalled tensorboard-2.5.0\n",
            "  Attempting uninstall: gast\n",
            "    Found existing installation: gast 0.4.0\n",
            "    Uninstalling gast-0.4.0:\n",
            "      Successfully uninstalled gast-0.4.0\n",
            "  Attempting uninstall: tensorflow\n",
            "    Found existing installation: tensorflow 2.5.0\n",
            "    Uninstalling tensorflow-2.5.0:\n",
            "      Successfully uninstalled tensorflow-2.5.0\n",
            "  Running setup.py develop for nmt-keras\n",
            "Successfully installed gast-0.2.2 h5py-2.10.0 keras-2.3.1.1 keras-applications-1.0.8 multimodal-keras-wrapper-3.1.6 nmt-keras-0.6 numpy-1.18.5 portalocker-2.0.0 sacrebleu-1.5.1 sacremoses-0.0.45 subword-nmt-0.3.7 tensorboard-1.15.0 tensorflow-1.15.5 tensorflow-estimator-1.15.1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "numpy"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ndy2AL3ZJwO2"
      },
      "source": [
        "Let's import necessary modules:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fFsrTRFAJwdY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0e9ec04b-8142-435c-d9a2-feef694bd09d"
      },
      "source": [
        "from keras.layers import *\n",
        "from keras.models import model_from_json, Model\n",
        "from keras.optimizers import Adam, RMSprop, Nadam, Adadelta, SGD, Adagrad, Adamax\n",
        "from keras.regularizers import l2\n",
        "from keras_wrapper.cnn_model import Model_Wrapper\n",
        "from keras_wrapper.extra.regularize import Regularize\n"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n",
            "[30/07/2021 09:28:20] NumExpr defaulting to 2 threads.\n",
            "[30/07/2021 09:28:20] <<< Cupy not available. Using numpy. >>>\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6VFrgZAaJ4Ke"
      },
      "source": [
        "And let's define the dimesnions of our model. For instance, a word embedding size of 50 and 100 units in RNNs. The inputs/outpus are defined as [the NMT-Keras tutorial](https://colab.research.google.com/github/lvapeab/nmt-keras/blob/master/examples/tutorial.ipynb/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eKsl10mIKKT5"
      },
      "source": [
        "ids_inputs = ['source_text', 'state_below']\n",
        "ids_outputs = ['target_text']\n",
        "word_embedding_size = 50\n",
        "hidden_state_size = 100\n",
        "input_vocabulary_size=686  # Autoset in the library\n",
        "output_vocabulary_size=513  # Autoset in the library\n"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xyJjZ9WiKMSn"
      },
      "source": [
        "## Encoder\n",
        "\n",
        "Let's define our encoder. First, we have to create an Input layer to connect the input text to our model.  Next, we'll apply a word embedding to the sequence of input indices. This word embedding will feed a Bidirectional GRU network, which will produce our sequence of annotations:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w_56taSDKSvJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bed8d40a-f4ab-4768-c5d3-ee002c713a7e"
      },
      "source": [
        "\n",
        "# 1. Source text input\n",
        "src_text = Input(name=ids_inputs[0],\n",
        "                 batch_shape=tuple([None, None]), # Since the input sequences have variable-length, we do not retrict the Input shape\n",
        "                 dtype='int32')\n",
        "# 2. Encoder\n",
        "# 2.1. Source word embedding\n",
        "src_embedding = Embedding(input_vocabulary_size, word_embedding_size, \n",
        "                          name='source_word_embedding', mask_zero=True # Zeroes as mask\n",
        "                          )(src_text)\n",
        "# 2.2. BRNN encoder (GRU/LSTM)\n",
        "annotations = Bidirectional(GRU(hidden_state_size, \n",
        "                                return_sequences=True  # Return the full sequence\n",
        "                                ),\n",
        "                            name='bidirectional_encoder',\n",
        "                            merge_mode='concat')(src_embedding)\n"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:650: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:28:20] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:650: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:4786: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:28:20] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:4786: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:157: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:28:20] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:157: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:3599: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:28:21] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:3599: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I1lvOWzAKanL"
      },
      "source": [
        "## Decoder\n",
        "Once we have built the encoder, let's build our decoder.  First, we have an additional input: The previously generated word (the so-called state_below). We introduce it by means of an Input layer and a (target language) word embedding:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KxF2gpx_Kc5P"
      },
      "source": [
        "# 3. Decoder\n",
        "# 3.1.1. Previously generated words as inputs for training -> Teacher forcing\n",
        "next_words = Input(name=ids_inputs[1], batch_shape=tuple([None, None]), dtype='int32')\n",
        "# 3.1.2. Target word embedding\n",
        "state_below = Embedding(output_vocabulary_size, word_embedding_size,\n",
        "                        name='target_word_embedding', \n",
        "                        mask_zero=True)(next_words)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ltygQnKuKfG1"
      },
      "source": [
        "The initial hidden state of the decoder's GRU is initialized by means of a MLP (in this case, single-layered) from the average of the annotations:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pevqAIabKhcs"
      },
      "source": [
        "ctx_mean = MaskedMean()(annotations)\n",
        "annotations = MaskLayer()(annotations)  # We may want the padded annotations\n",
        "\n",
        "initial_state = Dense(hidden_state_size, name='initial_state',\n",
        "                      activation='tanh')(ctx_mean)\n"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Gma1o3eKj-U"
      },
      "source": [
        "So, we have the input of our decoder:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DtSXJMFKKmaO"
      },
      "source": [
        "input_attentional_decoder = [state_below, annotations, initial_state]\n"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eha8Tkc5KoaP"
      },
      "source": [
        "Note that, for a sample, the sequence of annotations and initial state is the same, independently of the decoding time-step. In order to avoid computation time, we build two models, one for training and the other one for sampling. They will share weights, but the sampling model will be made up of  two different models. One (model_init) will compute the sequence of annotations and initial_state. The other model (model_next) will compute a single recurrent step, given the sequence of annotations, the previous hidden state and the generated words up to this moment. \n",
        "\n",
        "Therefore, now we slightly change the form of declaring layers. We must share layers between the decoding models. \n",
        "\n",
        "So, let's start by building the attentional-conditional GRU:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JSXShJxOKslb"
      },
      "source": [
        "\n",
        "# Define the AttGRUCond function\n",
        "sharedAttGRUCond = AttGRUCond(hidden_state_size,\n",
        "                              return_sequences=True,\n",
        "                              return_extra_variables=True, # Return attended input and attenton weights \n",
        "                              return_states=True # Returns the sequence of hidden states (see discussion above)\n",
        "                              )\n",
        "[proj_h, x_att, alphas, h_state] = sharedAttGRUCond(input_attentional_decoder) # Apply shared_AttnGRUCond to our input\n"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cBRXE2k3Kty-"
      },
      "source": [
        "Now, we set skip connections between input and output layer. Note that, since we have a temporal dimension because of the RNN decoder, we must apply the layers in a TimeDistributed way. Finally, we will merge all skip-connections and apply a 'tanh' no-linearlity:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jbnSTz1QKwmz"
      },
      "source": [
        "\n",
        "# Define layer function\n",
        "shared_FC_mlp = TimeDistributed(Dense(word_embedding_size, activation='linear',),\n",
        "                                name='logit_lstm')\n",
        "# Apply layer function\n",
        "out_layer_mlp = shared_FC_mlp(proj_h)\n",
        "\n",
        "# Define layer function\n",
        "shared_FC_ctx = TimeDistributed(Dense(word_embedding_size, activation='linear'),\n",
        "                                name='logit_ctx')\n",
        "# Apply layer function\n",
        "out_layer_ctx = shared_FC_ctx(x_att)\n",
        "shared_Lambda_Permute = PermuteGeneral((1, 0, 2))\n",
        "out_layer_ctx = shared_Lambda_Permute(out_layer_ctx)\n",
        "\n",
        "# Define layer function\n",
        "shared_FC_emb = TimeDistributed(Dense(word_embedding_size, activation='linear'),\n",
        "                                name='logit_emb')\n",
        "# Apply layer function\n",
        "out_layer_emb = shared_FC_emb(state_below)\n",
        "\n",
        "shared_additional_output_merge = Add(name='additional_input')\n",
        "additional_output = shared_additional_output_merge([out_layer_mlp, out_layer_ctx, out_layer_emb])\n",
        "shared_activation_tanh = Activation('tanh')\n",
        "out_layer = shared_activation_tanh(additional_output)\n"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MIWAqThKKz5n"
      },
      "source": [
        "Now, we'll' apply a deep output layer, with linear activation:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b0HSKUUQK1g4"
      },
      "source": [
        "\n",
        "shared_deep_out = TimeDistributed(Dense(word_embedding_size, activation='linear', name='maxout_layer'))\n",
        "out_layer = shared_deep_out(out_layer)\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dojVKD6yK208"
      },
      "source": [
        "Our output layer is a projection to the target vocabulary space plus a softmax function for obtaining a probability distribution over the target vocabulary words at each timestep.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kZStpeXGK4Tg"
      },
      "source": [
        "shared_FC_soft = TimeDistributed(Dense(output_vocabulary_size,\n",
        "                                               activation='softmax',\n",
        "                                               name='softmax_layer'),\n",
        "                                         name=ids_outputs[0])\n",
        "softout = shared_FC_soft(out_layer)\n"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aXi4oN03K7yM"
      },
      "source": [
        "And we build the Keras model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qg4NpRNiLMv3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "adecab20-57cb-486f-f2d9-6b6e093270b2"
      },
      "source": [
        "model = Model(name='NMT Model', inputs=[src_text, next_words], outputs=softout)\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"NMT Model\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "source_text (InputLayer)        (None, None)         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "source_word_embedding (Embeddin (None, None, 50)     34300       source_text[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "bidirectional_encoder (Bidirect (None, None, 200)    90600       source_word_embedding[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "state_below (InputLayer)        (None, None)         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "masked_mean_1 (MaskedMean)      (None, 200)          0           bidirectional_encoder[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "target_word_embedding (Embeddin (None, None, 50)     25650       state_below[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "mask_layer_1 (MaskLayer)        (None, None, 200)    0           bidirectional_encoder[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "initial_state (Dense)           (None, 100)          20100       masked_mean_1[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "att_gru_cond_1 (AttGRUCond)     [(None, None, 100),  135501      target_word_embedding[0][0]      \n",
            "                                                                 mask_layer_1[0][0]               \n",
            "                                                                 initial_state[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "logit_ctx (TimeDistributed)     (None, None, 50)     10050       att_gru_cond_1[0][1]             \n",
            "__________________________________________________________________________________________________\n",
            "logit_lstm (TimeDistributed)    (None, None, 50)     5050        att_gru_cond_1[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "permute_general_1 (PermuteGener (None, None, 50)     0           logit_ctx[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "logit_emb (TimeDistributed)     (None, None, 50)     2550        target_word_embedding[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "additional_input (Add)          (None, None, 50)     0           logit_lstm[0][0]                 \n",
            "                                                                 permute_general_1[0][0]          \n",
            "                                                                 logit_emb[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "activation_1 (Activation)       (None, None, 50)     0           additional_input[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "time_distributed_1 (TimeDistrib (None, None, 50)     2550        activation_1[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "target_text (TimeDistributed)   (None, None, 513)    26163       time_distributed_1[0][0]         \n",
            "==================================================================================================\n",
            "Total params: 352,514\n",
            "Trainable params: 352,514\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E8jGtjbVLX2O"
      },
      "source": [
        "\n",
        "That's all! We built a NMT Model!\n",
        "\n",
        "## Sampling models\n",
        "\n",
        "Now, let's build the models required for sampling. Recall that we are building two models, one for encoding the inputs and the other one for advancing steps in the decoding stage.\n",
        "\n",
        "Let's start with model_init. It will take the usual inputs (src_text and state_below) and will output: \n",
        "1)  The vector probabilities (for timestep 1)\n",
        "2) The sequence of annotations (from encoder)\n",
        "3) The current decoder's hidden state\n",
        "\n",
        "The only restriction here is that the first output must be the output layer (probabilities) of the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ujV4BlW_LeIn"
      },
      "source": [
        "\n",
        "model_init = Model(inputs=[src_text, next_words], outputs=[softout, annotations, h_state])\n",
        "# Store inputs and outputs names for model_init\n",
        "ids_inputs_init = ids_inputs\n",
        "\n",
        "# first output must be the output probs.\n",
        "ids_outputs_init = ids_outputs + ['preprocessed_input', 'next_state']"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YdHVfRWdLkz4"
      },
      "source": [
        "Next, we will be the model_next. It will have the following inputs:\n",
        "* Preprocessed input\n",
        "* Previously generated word\n",
        "* Previous hidden state\n",
        "\n",
        "And the following outputs:\n",
        "* Model probabilities\n",
        "* Current hidden state\n",
        "\n",
        "So first, we define the inputs:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M6yZRsNLLnM1"
      },
      "source": [
        "\n",
        "preprocessed_size = hidden_state_size*2\n",
        "preprocessed_annotations = Input(name='preprocessed_input', shape=tuple([None, preprocessed_size]))\n",
        "prev_h_state = Input(name='prev_state', shape=tuple([hidden_state_size]))\n",
        "input_attentional_decoder = [state_below, preprocessed_annotations, prev_h_state]"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "noPxKdghLpup"
      },
      "source": [
        "And now, we build the model, using the functions stored in the 'shared*'  variables declared before:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7cBQ1_mULsLf"
      },
      "source": [
        "# Apply decoder\n",
        "[proj_h, x_att, alphas, h_state] = sharedAttGRUCond(input_attentional_decoder)\n",
        "out_layer_mlp = shared_FC_mlp(proj_h)\n",
        "out_layer_ctx = shared_FC_ctx(x_att)\n",
        "out_layer_ctx = shared_Lambda_Permute(out_layer_ctx)\n",
        "out_layer_emb = shared_FC_emb(state_below)\n",
        "additional_output = shared_additional_output_merge([out_layer_mlp, out_layer_ctx, out_layer_emb])\n",
        "out_layer = shared_activation_tanh(additional_output)\n",
        "out_layer = shared_deep_out(out_layer)\n",
        "softout = shared_FC_soft(out_layer)\n",
        "model_next = Model(inputs=[next_words, preprocessed_annotations, prev_h_state],\n",
        "                   outputs=[softout, preprocessed_annotations, h_state])"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QFU5I-u0LtuM"
      },
      "source": [
        "Finally, we store inputs/outputs for model_next. In addition, we create a couple of dictionaries, matching inputs/outputs from the different models (model_init->model_next, model_nex->model_next):\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cbSc5mFQLvJd"
      },
      "source": [
        "# Store inputs and outputs names for model_next\n",
        "# first input must be previous word\n",
        "ids_inputs_next = [ids_inputs[1]] + ['preprocessed_input', 'prev_state']\n",
        "# first output must be the output probs.\n",
        "ids_outputs_next = ids_outputs + ['preprocessed_input', 'next_state']\n",
        "\n",
        "# Input -> Output matchings from model_init to model_next and from model_next to model_nextxt\n",
        "matchings_init_to_next = {'preprocessed_input': 'preprocessed_input', 'next_state': 'prev_state'}\n",
        "matchings_next_to_next = {'preprocessed_input': 'preprocessed_input', 'next_state': 'prev_state'}\n"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QAHr0oLQLwRq"
      },
      "source": [
        "And that's all! For using this model together with the facilities provided by the [multimodal_model_wrapper](https://github.com/MarcBS/multimodal_keras_wrapper) library, we should declare the model as a method of a Model_Wrapper class. A complete example of this can be found at [`model_zoo.py`](https://github.com/lvapeab/nmt-keras/blob/master/nmt_keras/model_zoo.py).\n"
      ]
    }
  ]
}